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ABSTRACT 

This study examined the effects of three variables on 
self-scoring accuracy and satisfaction vith results on Holland's Self 
Directed Search. The sample consisted of 489 freshmen who took the 
SDS during orientation. The three independent variables were the test 
administrator's attitude toward ihe SDS (positive or neutral), the 
size of the group taking the SDS (25 or 100), and the use of monitors 
during the administration (monitoring or no monitoring). Over 
one-fourth of the subjects made scoring errors resulting in incorrect 
high point codes, and over one-half made errors affecting their final 
three-letter summary code. Less than half the subjects felt their 
results were useful or reasonable. Of the three independent 
variables, only monitoring significantly reduced self-scoring errors, 
and none affected satisfaction, although trends toward interactions 
of the two other variables with monitoring emerged. Questions were 
raised about whether, even with monitoring, error rates are too high 
and satisfaction too low to warrant use of the SDS, in its present 
form, as a self-counseling device. (Author) 
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There has been a long-standing need in vocational counseling for a short, 
self-administering, and self-interpreting instrument that efficiently sum- 
marizes an individual's characteristics and links them to relevant occupations. 
Ideally, such an instrument would eliminate the time lag involved in machine 
scoring and obviate much of the need for face-to-face counseling. An instru- 
ment of this sort recently has been developed by Holland (1971), and prelimi- 
nary research suggests that simply completing it (including self-interpretation) 
has a beneficial effect til students 1 vocational development (Zener & Schnuelle, 
1972). 

The Self -Directed S earch (SDS) is a self -administered , self-scored, and 
self-interpreted instrument designed to ... "provide a vocational counseling 
experience for people who do not have access to professional counselors, or 
who cannot afford their services, and to multiply the number of people a coun- 
selor can serve (Holland, 1972, p. 3)." It is based on Holland's (1966) 
theory of vocational choice. The theory asserts that vocational choice is a 
reflection of personality, and that there are six vocationally-relevant per- 
sonality types: realistic, investigative, social, artistic, conventional, 
enterprising. The SDS enables people to assess their resemblance to each of 
the six theoretical personality types and to determine the resulting occupa- 
tions for which they would be most suited (Holland, Viernstein, Kuo, Karweit & 
Blum, 1970). 

Despite its potential usefulness, an investigation by Gelso, Collins, 
Williams and Sedlacek (1973) revealed that a high percentage of incoming college 
freshmen make serious scoring errors on the SDS (as many as 50%). In addition, 
less than half the subjects felt their SDS results were reasonable. 

Since the notion of a self-administering guidance system does have much 
merit (Holland, 1971, 1972), and because at least two studies support the 
value of the SDS in vocational guidance (Christer.sen & Sedlacek, 1974; Zener & 
Schnuelle, 1972), the authors considered it important to search for ways of 
reducing scoring errors (which attenuate both reliability and validity) and 
of enhancing students 1 attitudes toward the instrument. Rather than reorga- 
nizing the instrument itself, it was viewed as more parsimonious to begin 
such a search by studying variables in the testing situation. If such variables 
were found to affect scoring accuracy and satisfaction with the SDS, they 
could then be manipulated accordingly in subsequent test administrations. 
Research conducted on the SDS indirectly suggests that two factors may affect 
scoring accuracy: the size of the group taking the instrument and whether 
or not the administration is carefully monitored (Baldwin, 1971; Gelso, 
et al. 1973; Holland, 1972; O f Connell & Sedlacek, 1971; Zener & Schnuelle, 
1972). Additionally, earlier research suggests that a test administrator's 
attitude toward the instrument being administered may affect subjects 1 per- 
formance on and feelings about the test (Masling, 1960; Weiner, 1957). Thus, 
the present experiment sought to determine the effect of three variables (ad- 
ministrator attitude toward the SDS, group size, monitoring of the test ad- 
ministration) on both the accuracy of students 1 self-scoring and students 1 
satisfaction with their SDS results. 



2 



Method 



Sample and Procedure 

As part of the 1972 summer orientation program at the University of 
Maryland, 489 incoming freshmen completed the SDS. When students arrived for 
testing they were randomly assigned to one of eight subgroups in which they 
completed the instrument. Each subgroup was given the SDS by a test ad- 
ministrator, whose role it was to introduce briefly the instrument. Test- 
taking lasted a maximum of 60 minutes. 

The eight subgroups to which students were assigned represented combina- 
tions of the three independent variables: administrator attitude toward the 
SDS (positive or neutral), group size (25 or 100), and whether or not the test 
administration was monitored. In its final form the experimental design was 
a 2 x 2 x 2 crossed and balanced factorial design. 

Administrator attitude (positive or neutral) was reflected in the ad- 
ministrator's introduction to the SDS as follows: 

Positive attitude : In a few minutes I will be passing out a test which 
has been found to be extremely useful in vocational guidance. It is called 
the Self-Directed Search. As the name implies, you do your own scoring, pro- 
filing and interpreting of your results. You will obtain the results immediate- 
ly and relate them to 414 possible occupations. For some of you, vocational 
and academic planning are closely associated. If you don't know what to study 
or what you want to do for a living, it can be a problem. This test may be 
of great help to those of you who find yourselves in this situation. If you 
are reasonably certain about your vocational goals, the results of this test 
will help in reaffirming your ideas. 

Neutral attitude : In a few minutes I will be passing out a test called 
the Self-Directed Search. It is used in vocational guidance. As the name 
implies, you do your own scoring, profiling and interpreting of your results. 
This test may be helpful to you in vocational and academic planning. 

The administrations were conducted over a two day period and were managed 
by two experienced t~st administrators (graduate students in counseling). 
Each administrator engaged in an equal number of administrations undet each 
experimental condition. 

Whether or not test administrations were monitored was the second inde- 
pendent variable. The four groups with monitoring had one monitor per 25 
students, following Zener and Schnuelle f s (1972) procedure. Monitors were 
undergraduar.es who were experienced in test monitoring and highly familiar 
with the SDS. They were instructed to offer assistance to students who seemed 
perplexed when taking the test and to answer any questions which ml '$ht arise. 
In the four non-monitored groups, monitors left the room after distributing 
the test booklets and returned in 60 minutes. 
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The third independent variable was group size. Jt was fixed at two levels, 
with one-half the groups containing 25 students and the remaining groups con- 
sisting of 100 students. It was expected that group size would interact with 
the presence or absence of monitors. Holland (1973) has noted that the SDS 
is effectively taken without monitoring in groups of 20 or 30, but that moni- 
tors may be needed with large groups. 

The SDS: Self-Administration and Scoring 

The SDS consists of two booklets, an Assessment Booklet and an Occupations 
Finder. The subject first completes the Assessment Booklet, which yields a 
three-letter summary code indicating the three personality types (from Holland's 
six types) to which he/she is most similar. The subject then consults the 
Occupations Finder for occupations most compatible with the code. 

Completion of the Assessment Booklet entails several steps. In summary, 
subjects are required to provide self-ratings of interests and competencies 
in numbers of activities, occupations, and general interest areas subsumed 
under Holland's personality typology. In the process of completing the Booklet, 
subjects are required to sum their scores at several points, transfer these 
sums to graphs and tables, and multiply their sums such that the end result 
is the aforementioned summary code composed of the three personality types 
(in order of prominence) the subject most closely resembles. 



Dependent Variables 

The dependent variables were (a) the accuracy of subjects 1 self- 
administration/scoring, and (b) subjects 1 satisfaction with the SDS. While 
there are a number of types of errors subjects may commit in taking the SDS 
(see Gelso, et al., 1973), only those which altered the three letter summary 
code (letter incorrectly omitted, letters in wrong order) or produced an in- 
correct high point code were considered in this study, since these are the 
most serious types of errors from a counseling standpoint. Error rates were 
analyzed by three experienced undergraduate research assistants. Two assistants 
each re-scored one-half the Assessment Booklets to detect errors that affected 
the summary code or high point code. Then, to assure that errors were not 
committed in the re-scoring itself, a third assistant re-scored all assessment 
booklets again. Thus, all booklets were re-scored twice. When the two re- 
scorings and noting of errors did not coincide (about 20% of the cases), the 
Testing Office f s psychome trist , a person highly familiar with the SDS, resolved 
the difference by again re-scoring the: booklet. 



Subjects 1 satisfaction with the SDS was measured by two items that have 
been used in several SDS studies (Collins & Sedlacek, 1972; Gelso, et al., 
1973; Kimball, Sedlacel & Brooks, 1973). After taking the SDS, subjects 
responded to the statements: (a) "My summary code occupations seem reasonable 
for me; ' (b) "1 think the results will be useful to me in future academic 
planning." Each item was responded to on a 5-point I.tkurt scale as follows: 
1 - strongly agree, 2 = agree, 3 = neutral, 4 = disagree, 5 - strongly disagree. 
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Results 

To balance the experimental design, it was planned to select randomly 25 
booklets from each of '.he four groups with n f s of 100. Only 23 students in 
two of the four groups with pre-planned n f s of 25 attended the testing sessions. 
Thus, to keep balancing intact, 23 Assessment Booklets were analyzed for each 
of the eight groups, resulting in an overall n of 184. 

Krror Rates 

Overall, 108 of the 184 (59%) students made errors that altered their 
three-letter summary code, and 48 (26%) made mistakes which produced erroneous 
high point codes. Chi square analyses were performed on the frequency of cases 
in which errors were made that affected high point codes and summary codes 
according to each experimental variable (group size, administrator attitude, 
monitoring). Frequency data are presented on Table 1. It can be seen that 
very similar error rates occurred when the group size was 25 and 100, and when 
the test administrator's attitude was positive and neutral. The effects of 
these two variables did not approach the .05 level of significance. The moni- 
toring variable, however, did affect significantly errors in high point codes. 
Thirty-one of the 92 students (34%) in the four non-monitored groups had 
erroneous high point codes, while only 17 of the 92 students (18%) in the four 
monitored groups had inaccurate high point codes (X* = 5.53, p < .02). While 
a similar pattern emerged for the effects of the monitoring variable on summary 
code errors, it did not approach statistical significance (X* - 1.44, p = .23). 



Insert Table 1 About Here 



To determine the interaction effects of the three independent variables 
on the two types of errors, 2x2x2 analyses of variance were performed, 
viewing the dependent variables as two-point continua, i.e., error ■ 1, no 
error =2. No significant interactions emerged for high point or summary code 
errors, although a trend (F ■ 2.91, p * .09) existed toward a monitoring by 
administrator attitude interaction. Students in the non-monitored groups that 
were presented with positive administrator attitudes toward the SDS exhibited 
a tendency toward higher error rates than those in the other three monitoring 
X administrator attitude treatment combinations. 

Satisfaction with the SDS 

In response to the statement, "My summary code occupations seem reason- 
able to me," the following frequencies emerged for all 184 students: Strongly 
agree = 19 (10%), agree = 64 (35%) . neutral = 54 (29%), disagree = 25 (14%), 
strongly disagree * 22 (12%). To determine the effects of group size, ad- 
ministrator attitude, and monitoring on students' responses to this item, a 
2x2x2 analysis of variance was performed. None of the main or interaction 
effects attained statistical significance, although there was a nonsignificant 
tendency (F = 2.49, p * .11) for students who took the SDS under monitored 
conditions to respond more favorably than those whose test-taking was non- 
monitored (X f s = 2.69 vs. 2.96 in the monitored and non-monitored conditions 
respectively; lower scores indicate great agreement). 
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The statement, "I think the results will be useful to me in my future 
academic planning, 11 elicited the following responses: Strongly agree = 11 
(6%), agree = 58 (32%), neutral = 70 (382), disagree = 23 (13%), strongly disa- 
gree = 22 (12%). The three-way analysis of variance on tills item indicated 
that no main or interaction effects attained statistical significance. Again, 
however, there was a trend (F = 2.97, p = .09) in the direction of students 
in the monitored condition responding more positively than those in the non- 
monitored condition (X f s = 2.79 vs. 3.07 in the monitored vs. non-mor itored 
condition respectively, with lower scores indicating greater agreement). Also, 
a trend (£ = 2.98, p « .09) emerged toward a monitoring by group size inter- 
action, with students in the smaller groups (n = 25) and monitored condition 
responding more positively than those in the other three group size-monitoring 
treatment combinations . 

Discussion 

In general, the data suggest that a large percentage of students make 
serious errors when completing the SDS. Over one -fourth in the present study 
made errors which produced incorrect high point codes and more than half made 
mistakes that resulted in inaccurate summary codes. In addition, subjects 
did not appear to be particularly satisfied with the instrument. These find- 
ings are highly consistent with those of Gelso, et al., (1973), the study which 
formed the basis for the present experiment. 

The results do indicate that careful monitoring of SDS test taking is 
helpful in reducing scoring errors. Significantly fewer protocals had errone- 
ous high point codes when test taking was monitored than when it was not (18% 
vs. 34% respectively). Also, there was a consistent trend toward monitoring 
having a desirable effect on the remaining dependent variables. These results 
support Holland's (1972) belief that monitors are needed when the SDS is given 
in larger groups. The findings also indicate, however, that monitors are help- 
ful in smaller groups. Thus, it may be important to use trained monitors in 
groups of any size. This sort of finding raises serious questions about whether 
the SDS, as it currently stands, can be truly "self directed 11 , or at least 
as self directed as its constructor has purported it to be. 

Neither of the two remaining independent variables, group size and ad- 
ministrator attitude toward the SDS, were found to affect either scoring accu- 
racy or satisfaction, although trends toward their interacting with the moni- 
toring variable were uncovered. A word of caution is in order regarding the 
interpretation of these negative results. With respect to administrator 
attitude, it must be remembered that the two levels chosen for this experiment 
were "positive" and "neutral." Thus, the results in no way imply that a 
negative attitude by the administrator would not dampen students 1 attitudes 
toward the instrument. Similarly, the results do not imply that scoring accu- 
racy and/or satisfaction would not be attenuated by administration to groups 
considerably larger than 100. Since a fixed effects model was employed in 
this experiment, caution must be exercised in generalizing much beyond the 
levels that were chosen for each variable. 

As implied, subjects did not appear to experience a high degree of satis- 
faction with their SDS results. Only 38Z agreed or strongly agreed that their 
results would be useful in future academic planning, and only 45% similarly 



agreed that the occupations suggested by their summary code seemed reasonable. 
It will be recalled that students in the present study took the SDS during 
freshman orientation. Might this indicate that we are dealing with unmoti- 
vated subjects who, due to low motivation, have negative attitudes toward the 
test-taking experience and make many errors when completing the SDS? The 
results of Gelso, et al., (1973) do not support such a speculation. They found 
that most students in a highly similar situation claimed to be very interested 
in learning more about what academic majors and occupations they migh^ like. 
More to the point, students' degree of interest was not found to be a factor 
in their evaluation of the reasonableness of their summary code occupations, 
or in scoring accuracy for that matter. Taken together, the evidence suggests 
that the SDS may be less attractive to university students than the author 
of the instrument has suspected (see Holland, 1971, p. 175). 

At a minimum, the data imply that when the SDS is taken by incoming 
freshmen during orientation, the error rates are appreciably higher, and 
students 1 satisfaction with their results is much lower, than would seem 
desirable. Careful monitoring does reduce scoring errors. But does such 
monitoring reduce errors enough to warrant use of the instrument as a "self- 
counseling" tool? While admitting that an answer to this question must be 
relative and somewhat arbitrary, the authors are inclined to answer in the 
negative. What appears to be called for now is a re-organization of the 
Assessment Booklet and a simplification of self-scoring procedures. Such 
revisions need to be assessed carefully, in terms of scoring accuracy at a 
minimum , before being utilized on a large scale with client populations. 
The University of Maryland Counseling Center is now exploring the use of a 
streamlined version of the SDS in combination with at least an initial inter- 
view with a client in which a judgment is made on whether the SDS is a suit- 
able treatment. Careful checks are being made on both scoring accuracy and 
clients 1 satisfaction with the SDS experience. 
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Table 1 

Error Rates in High Point Codes and Three-letter Summary Codes 

for Each Experimental Variable 
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Group Size 




100 1 


22 


70 


54 38 






25 


26 


66 


54 38 


Administrator 


Attitude 


Positive 


20 


72 


52 40 






Neutral 


28 


64 


56 36 


Monitoring 




Monitors 


17 


75 2 


50 42 






No Monitors 


31 


61 


58 34 


^"Analysis was 


performed 


using n f s of 23 


for each of 


the eight 


subgroups for the 



group size variable as well as the Administrator Attitude and Monitoring 
Variables. 

2 % 2 = 5.53, p ^ .02; all other * 2f s ns (p ' .05). 



